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ABSTRACT 

A model of evaluation planning, implementation, anc3 
management is presented that includes: (1) a management information 
system (MIS) to provide a description of the size and scope of adult 
literacy and post-li teracy programs (ALPPs) ; (2) naturalistic 
evaluation to provide insight into the meanings of ALPPs as 
experienced t)y participants; and (3) rationalistic evaluation of 
ALPPs. The model considers the MIS as the foundation stone of a 
dynamic culture of information. Naturalistic evaluation adds meaning 
to the data. Rationalistic evaluation is used to: make normative 
statements, make comparisons between groups, and assert correlations 
between particular variables. Implementing this model in 
international set :ings requires: commitment to internal evaluation; 
introduction of the model in settings providing technical assistance 
and recognition of the outside evaluator's role as a skilled guesL, 
qualified in the theory and methodology of evaluation. Meta 
evaluative objectives and the input of all stakeholders will be 
required to achieve a dynamic culture of information, reviewing both 
instrumental and moral questions. One figure illustrates the model. 
(SLD) 
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MANAGEMENT INFORMATION SYSTEMS FOR BASIC EDUCATION: 
DISCOVERING AND SUPPORTING CURRENT BEST PRACTICE 
IN ADULT LITERACY AND POST-LITERACY EVALUATION 



H.S. Bhola 



This paper will focus on management information systems 
in the area of adult literacy and post-literacy. However, the 
focus on management information systems will not mean an 
exclusion of all other evaluation approaches to information 
gathering and evaluation. Indeed, the model of evaluation 
planning, implementation and management, to be presented in the 
body of the paper, is a triangle of approaches that includes (i) 
a management information system (MIS) to provide a description of 
the size and scope of literacy and post-literacy programs; (ii) 
r.=ituralistic evaluation (NE) to provide insights into the 
meanings of literacy and post-literacy as experienced by 
participants as individuals and their communities; and (iii) 
rationalistic evaluation (RE) to enable practitioners to make 
normative statements about the total program, to render 
compari£;ons and discover correlations between variables and 
aspects of programs, if and when needs arise. 

The paper also takes some clear value positions. First, 
we hold the value that evaluation should be, first and foremost, 
jjOLtsenai evaluation, to serve the purposes of the program and the 
interests of the participants in a program. External uses of the 
evaluation data and results should be secondary. The second 
value that pervades all discussion that follows is that of 
collaboration and participation among all stakeholders. Third, 
it is being suggested that in installing management information 
systems (and in implementing evaluation studies) , the value of 
discovering and supporting current best practice should be 
honestly embraced. That is, the indigenous and the already 
acquired modes and patterns of gathering, generating, 
synthesizing, legitimizing and interpreting information — both 
descriptive and evaluative — for use in decision making should 
be understood; and these modes and patterns of information 
handling should become the foundations for a renewed culture of 
information and a future system of evaluation. 

BETWEEN THEORY AND PRACTICE: A PERSONAL JOURNEY 

I will not be saying anything unusual if I stated that my 
individual construct of evaluation, and the model I will be 
presenting later, have come ab'"ut through my own personally- 
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experienced dialectic between evaluation theory and evaluation 
practice. The theories of evaluation were learned from the 
universities (mostly, the Ohio State University) , and 
professional associations (principally, the American Evaluation 
Association) in the United states, over a period of time 
streching from 1964 to the present. These theories became 
meaningful in my life through personal contacts with my teacher, 
colleague and friend. Professor Egon G. Cuba, an eminent world- 
renowned theoretician of evaluation — which is not to sf».y that 
he will put his seal of approval on the model that I will be 
presenting below. 

Opportunities for the practice of evaluation became 
available to me, first, as a UNESCO field expert on the 
Experimental World Literacy Program in Tanzania during 1968-70; 
and, later, as part of consultancy missions in behalf of UNESCO, 
USAID and, much more consistently and frequently, with the German 
Foundation for International Development (DSE) , Bonn. These 
missions, during the last fifteen years took me t:o Tanzania, 
Kenya, Zambia, Malawi, Ethiopia, Botswana, Zimbabwe and China. 

LEARNING FROM EVALUATION THEORY 

My personal journey in evaluation theory is by no means 
unique to me. I was, of course, trained in positivist 
epistemology, logical-deductive methodology and statistical 
analysis at the Ohio state University in the early 1960s. But 
seeds of dcubt about the supremacy of the positivist paradigm 
were being sown in my mind even as I was learning the so-cailed 
"scientific" paradigm. Professors in classes talked glibly of 
continua, and systems. Kaplan (1964) sought to demystify the 
conduct of inquiry and assured us that inquiry indeed did not 
work the way positivists would have us believe. Berger and 
Luckmann (1966) talked of the social construction of reality. 
Campbell and Stanley (1966) were retreating, at least 
indirectly, from the experimental design as they talked about 
quasi-experimental design, while Webb and his associates (1966) 
lent respectability to unobstrusive measures. Glaser and 
Strauss (1967) made the concept of grounded theory popular. 
People were beginning to talk about ethnographies and case 
studies with deserved respect. The rest is history! 

In the area of evaluation theory, in particular, I 
started with the work of Egon G. Guba and Dan Stuff lebeam at 
Ohio State. Later, the pdk book on evaluation (1971) oecame a 
dependable source. The availability of work on taxonomies of 
educational objectives, and educational and social indicators 
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had led me to accept the usefulness of management information 
systems. I did become aware of the many models of evaluation 
— the discrepancy evaluation model, goal-free evaluation, 
transactional evaluation approach, evaluation as illumination, 
advocacy model of evaluation, and participatory evaluation; and 
I did seek to integrate all I had learned during the 1960s and 
1970s in my own book Evaluating Functional Literacy (Bhola, 
1979). The book did indeed talk of indicators and MIS's, and of 
both formalistic and naturalistic approaches, though these three 
components did not find the complete integration achieved in more 
recent work (Bhola, 1990, 1991). 

In the meantime, discussion of evaluation theory came to 
be conducted not in terms of evaluation models but in terms of 
paradigms of inquiry such as logical-positivism, 
constructionism, critical theory and so on as captured in Cuba 
(1990). Understandably, as the 1980s began, I had found myself 
leaning heavily towards naturalistic evaluation strategies. 
That seemed like the only sensible thing to do. But I did so 
without a complete rejection of everything else. I had not 
thrown away RE, nor, for that matter, the MIS. Indeed, the MIS 
became central to tny conception of evaluation plannii.7 
implementation and management. The world of practice nad 
cautioned against clean breaks and compelled accomjnodation among 
approaches . 

LEARNING FROM EVALUATION PRACTICZ 

Practical experience in evaluation came to me from my 
work in Tanzania, Kenya, Zambia, Botswana, Ethiopia, Malawi and 
Zimbabwe in Africa, and from India and China in Asia. It is not 
possible within the scope of this paper to review these 
experiences in any detail, and it seems unnecessary to include a 
list of relevant references, most of them unpublished reports 
that are not easily available. We will be satisfied by 
mentioning that irrespective of the theoretical discussion of 
the possibility or the impossibility of combining or reconciling 
different paradigms, the practitioners demanded accommodation 
between and among approaches. The wanted not theoretical 
elegance but all the various tools that would help them 
understand their programs and also help them survive the 
politics that pervaded the processes of program planning, 
implementation and its evaluation. 

The following were some concrete learnings from the world 
of practice: 
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a. In the Third World, the practitioners' conception of 
evaluation is what we have called rationalistic evaluation. 
This is what they have all been hearing about for years; and 
this is what is demanded of them by most donors from abroad, 
and, therefore, by their bosses at home. Breaking the magic of 
RE will not be easy* 

b. In view of the above, RE is what they want to learn about 
in training workshops and courses when theca are offered to them 
by donors under various technical assistance packages. 

c. Ironically, rationalistic evaluation is not the 
evaluation they actually conduct in raal-life contexts of their 
programs; and they hardly ever find themselves using results from 
this type of evaluatijn in doing their work. When an RE study is 
undertaken, it is typically conducted by an outsider; its results 
are available late and often not at all; and what results do 
become available are hardly ever used for programmatic purposes 
but are only put to political uses by policy makers. 

d. The inforration that practitioners do actually need, and 
do actually use when available, is descriptive information about 
the size and scope of their programs, and c bout changes in 
patterns of participation and effects over time. 

e. After, the descriptive informtion referred to in the 
immediately preceding paragraph, the information most used by 
practitioners is qualitative information picked up in field 
visits, informal interviews, conversations, and anecdotes. 

LESSONS FROM THE DIALECTIC BETWEEN THEORY AND PRACTICE 

From the dialectic between theory and practice, the 
lessons learned were the following; 

1- Neither bv positivism, nor bv constructionism alone . The 

first lesson was about the logical-positivist paradigm in 
general. It was clear that this was not the only mode of making 
assertions about the world. The related, and perhaps the more 
important, learning was that logical positivism had to be 
complemented with the constructionist paradigm. In the real 
world, there was need for both story and theory, for metaphor 
and mathematics, for numbers and meanings- 

2. Not merelv discre et and di scontinuous eval uation studies. 

but a culture of information. Too often evaluation in literacy 
and post-literacy projects, programs and campaigns gets 
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translated into occasional, discreet, and discontinuous, stand- 
alone evaluation studies in the so-called scientific or the 
natural i<5tic modes of inquiry. These evaluation studies consume 
a considerable amount of resources, generate lot of fear and 
tension among the various stakeholders, and their use is more 
often political than programmatic. They are often untimely. Of 
course, evaluations have to be done sometimes. At times, we need 
them, and sometimes we are ordered to do them. What we are 
suggesting here is that within organizations of literacy ant' 
post-literacy, we should seek to create "cultures of information" 
so that information collection, synthesis and utilization in 
decision-making becomes a norm and a habit. We are not 
suggesting a dichotomous choice: that we stop doing evaluation 
studies and get busy in creating cultures of information. We 
suggest that we give priority to developing cultures of 
information. Evaluation studies may be conducted within the 
context of such cultures of information. 

3 • Descrip tive information is oftener used than evaluative 

information. in the real world of planning and action, 
descriptive information (information on the size and *iCope of 
the program) is more often used than evaluative information 
(that makes comparisons, establishes correlations, or surveys 
attitudes, feelings or experiences) . Therefore, descriptive 
information of the type that goes into an MIS should be at the 
cjre of the strategy for bulding new cultures of information. 

4. The next most used information is qualitative 
infQr matjpn, b est obtained through naturalistic evaluation . In 
the world of practice, it is seen that after MIS, NE is the next 
most used mode of information generation. If systematic NE is 
not available, decision makers do infact go ahead and use 
impressionistic and anecdotal data. 

5. fiationaUstjc evaluation has a usef u l but infrequent role 
to play. Rationalistic evaluation used to be the only 
acceptable ^avaluation. There are, of course, several settings 
that cronbach characterises as "contexts of control" (quoted in 
Bhola, 1990, 1991) where RE models and approaches are the 
appropriate choices. But these situations are few and RE 
studies answer questions only regarding choices at the policy 
level and seldom at the programmatic level. 

6. N^edgc^ is ^ concrete model for develop i ng a culture of 
information. An ideal model for helping develop a culture of 
information should have an MIS system at the core, but it must 
also include NE, and RE. What is a dynamic culture of 
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information? Here is a description and definition: Any 
organization in the very process of being will acquire a culture 
of information, but what we need, of course, is a vibrant, 
democratic and smoothly functioning culture of information. 
Within an organizational context, such a "culture of information" 
would mean an institutional culture or sub-culture that 
systematically collects, generates, stores, retrieves, 
synthesizes, interprets and utilizes appropriate information in 
all processes of decision-making and does so as a matter of 
course. This would indeed mean that the organization 
systematically collects and stores descriptive data generated 
through the very process of xmplementation of its objectives and 
programs. It means also that such an organization would generate 
^v^XM^tive data through special evaluation studies appropriately 
designed and timed to be able to judge the value and worth of its 
various programs and projects. Information within such an 
organization will flow brck and forth horizontally; and will flow 
vertically across all levels, both up and down the system. 
Informed decision making will become a universa ily-shared norm. 

7- to discove r and s .u ppor t current best practice . All 

cultures, by definition, are also cultures ct information. 
Indeed, everywhere in the world, in the developing as well as in 
the developed world, information was collected and used to 
manage affairs of life, and multiple types of evaluations were 
made long before evaluation, as we know it today, came to the 
fore, acquired a scientific theory, and became a professional 
practice. Thus, the task is not to build anew but to renew. As 
we build renewed, nore systematic, more dynamic and more 
democratic cultures of information in various societies or 
within different professional sectors of societies, we must 
build "new" cultures of information on the "old" ones. Again, 
in teaching-learning new theories and methodologies of 
evaluation, we must discover and support the indigenous, 
already-in-use grounded theories and internalized methods of 
stocktaking, .-eceiving feedback, and assigning value and worth. 

Earticjpative strategies a re the only way to invent and 
^ctugljge a culture of information . Finally, we suggest that 
both m the process of inventing cultures of information and 
conducting evaluation studies, participative strategies be used. 
All through the process of building a culture of information, 
there is no room for "mere experts," because many of the choices 
are not merely technical choices but choices between values as 
well. Therefore, both for functional reasons and for ideological 
purposes, participative strategies are necessary. It is after 
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all, the participants alone who have knowledge of what is, and 
have the right to invent what ought to be. 

A MODEL FOR USE IN CREATING CULTURES OF INFORMATION 

A model to assist in developing a culture of information 
is presented below: 



[INSERT DIAGRAM HERE] 



Brief definitions of MIS, NE and RE were included in the 
opening paragraph of the paper. The theoretical contexts and 
the methodological implications of each have been discussed at 
several places in the paper. A few additional remarks are 
appropriate at this stage. 

THE MIS AS CORE: 

THE NECESSITY OF DESCRIPTIVE INFORMATION 

The model presented here considers the MIS as the 
foundation stone of a dynamic culture of information. It is too 
often forgotten that descriptive information, if not more 
important, is as least as important as evaluative information in 
management and decision making within program systems. By 
descriptive information we mean, of course, the information 
about the size and scope cf a program which is generated in the 
very process of implementation cf a program. Examples of such 
information are: the number of participants in a literacy or 
post-literacy program at various times in the program cycle, 
their distribution according to gender, age, and occupation, the 
number of groups and their locations, and the patterns of 
learner achievement and outcomes in regard to literacy, 
functionality and avareness. We should note that an MIS can 
include both quantitative and qualitative data and that such 
data can be manipulated to get evaluative information which 
otherwise would have to be obtained through specially designed 
and conducted evaluation studies. The mutual interdependences 
among MIS, NE and RE should also be noted. 

NATURALISTIC EVALUATION: 
ADDING MEANING TO NUMBERS 

Descx'iptive information about the scope and size of a 
program is pregnant with lot of evaluative information, but 
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additional evaluative information is often necessary • By 
evaluative information we mean information that enables program 
decision makers to make assertions in regard to (qualities, and 
meanings of programs; and to establish contrasts and 
correlations between components and aspects of program Inputs 
within particular contexts. T^e evaluative information provided 
particularly by NE (for RE see Delow) is thus information on the 
meanings of these programs as experienced by participants in 
their lives, NE is concerned with general patterns and 
dispositions, not theories and hypotheses. Methods are 
qualitative. Data analysis is thematic. Writing is in the form 
of case studies. The hope is to make warranted assertions in 
context and to be able to provide insights for other contexts. 

RATIONALISTIC EVALUATION: 

FOR COMPARISONS AND CORRELATIONS 

RE is best known around the world because it is best 
disseminated and has perhaps fulfilled people's needs for 
certainties and never-failing general principles. It assumes "a 
world out there for everyone to see" and about which law-like 
statements can be made. Its methods are experimental, and its 
analysis is typically statistical. In literacy and post- 
literacy, RE may be used for making normative statements about 
large populations^ to make comparisons between groups, and to 
assert correlations between particular variables. 

THE COLLABORATIVE-PARTICIPATIVE BIAS OF THE MODEL 

We should be cautious so as not to misconstrue the 
epistemic bias of the model as diagramed. The model does use 
the term MIS, a somewhat pretentious term made popular by 
computer-based system analysts. While we retained the term 
because it has become the current coin of discuo^ion in the 
evaluation field, we do not mean to imply that computers are 
necessary for developing MIS's. Paper-and-pencil MIS's can and 
sh<^uld be built where resources do not permit purchase of 
computer technology. Also, MIS*s should be built upon 
indigenous systems of management of information, howsoever 
rudimentary these existing systems may be. The model also 
favors NE over RE. After descriptive data provided by an MIS, 
NE is considered to be the most important mode of information 
development so as to be able to combine numbers with meanings. 
In favoring NE, we do not seek to promote an orthodoxy of theory 
or methodology. What is suggested are participative strategies 
for putting heads together to make collective constructions, 
leading to warranted assertions in particular contexts. Both the 
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intellectual and the intuitive should be used. The best of logic 
and of discernment are necessary. 

IMPLEMENTING THE ^lODEL IN INTERNATIONAL SETTINGS 

At first sight, the model presented above may not seem 
theoretically consistent and elegent. It may be viewed as built 
upon inconsistent assumptions and to include mixed categories of 
NE and RE both of whi:;h are predicated on an MIS, a technology 
with positivist ancestory. it will help to be reminded that the 
model is an instance of "grounded theory." in the real world, 
policy makers, planners, programmers and other practitioners of 
literacy and post-literacy have been found to need and work with 
both descriptive information and evaluative informtion. Further, 
in regard to evaluative information, practitioners have crying 
needs for understanding their program actions both in a normative 
sense (through RE) and in an experiential sense (through NE) . 
This model, as a child of grounded theory, rshould also be 
justifiable on grounds of higher epistemolo jical theory. While 
reality is an individual construction, we do come into "a world 
already half constructed." The world is indeed definable in the 
categories of "the context of accommodation" and "the context of 
control" (Chronbach quoted in Bhola, 1990, 1991) . Information 
ab^ut a phenomenon provided by the MIS, by NE and RE does add up 
to a richer picture. 

The ooTOitment to interna l evaluation , with leanings 
towards a constructionist view of reality, ve are in favor of 
collaborative strategies in the planning and delivery of 
technical assistance and participative approaches intra- 
nationally to do needs assessment, program development and 
evaluation at various levels. This model itself, we suggest, 
should be offered to colleagues within a collaborative framework 
as "something to think with," and it should be adapted for 
implementation in particular contexts within networks of 
participatory relationships among all stakeholders. 

, ^ Thg AntrgdMgtion of th e m odel in settings of techn^ gal 
assisiancg. We accept the necessity of teaching-learning across 
cultures and the need for flow of knowledge across national 
boundaries. We do, however, have a position of the nature of 
transactions of technical assistance around this model. The 
selling of this model should be seen not as a one-sided technical 
assistance from the donor to the recipient or unidirectional 
technology transfer or knowledge dissemination. The transaction 
of technical assistance should be seen as an articulation of a 
mutual commitment of donors and recipients; and some measure of 
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resource transfers from the donor to the recipient to help create 
an opportunity and an event around which evaluation of literacy 
programs within a system could take place on a regular basis. 

The role of the outsider . The role of the outsider (the 
bearer of the model in this case) should not be that of an all- 
knowing expert, consultant, or trainer, but as a colleague who 
has been around people who have gone through things of this 
sort. He or she should be seen as a good guest, who honors the 
recipient culture, is sympathetic to their aspirations, accepts 
them as what they are but who will yet honesty "hold the mirror 
to them" in which they may be able to see both their strengths 
and their shortcomings. 

The qualification s and skills of the cruest visitor - 
While the role of the visitor will not be that of an expert that 
does not mean that the visitor should have no expertise. 
Indeed, the visitor should have expertise in the theory and 
methodology of evaluation. The visitor should be able to 
understand both the instrumental and the moral role of 
evaluation. Most importantly, he or she should be able to look 
at the larger calculus of means and ends embedded in the program 
and the network of social relationships within which the program 
and its evaluation are being implemented. 

Since the style of interaction between the visitor and 
the visited will be dialogic, the visitor should understand the 
nature of human discourse, the dynamics of group processes, and 
should be able to promote genuine communication, in Edgar Dale's 
words, "sharing of meanings in a mood of mutuality." 

The meta objective in the intersvstem encounter . At the 
head of the list of concrete evaluation objectives, there should 
be meta evaluation objectives: 

1. to enable a particular group of stakeholders in a 
literacy or post-literacy program to step out of their 
immersions in the daily routines of the program and to look at 
the larger calculus of means and ends and the moral meanings of 
their actions; and 

2. to learn to do so self-consciously, using the best in (i) 
logic and (ii) discernment, rising above self-deceptions and 
misperceptions . 

DjggQVgrinq and supporting current best practice . The 
model to be implemented (and described in greater detail earlier 
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in the paper) is an empty set. It talks of MIS, NE and RE, but 
does not talk about what must be put in the empty set. Indeed, 
we suggest strongly that the model be re-invented within local 
settings and should be built on current best practice. 

What this would mean is that all stakeholders (including 
visitors) work collaboratively to identify and record the 
existing patterns of information generation, filing, recording, 
retrieving and using information. All systems will already have 
something in this regard, even though what exists may be 
inadequate or underutilized. What this means is that the new 
MIS should be designed and developed around the existing system 
of record keeping; the agenda for NE, and subsequently the 
methodologies of NE studies, should be based on the existing 
ways of making meanings out of field visits, community meetings, 
and whole set of unobstrusive measures. Finally, the agenda of 
RE itself should be based on existing procedures of conducting 
inspections, audits and accreditation visits. 

The achievement: a culture of information . If this 
encounter is successful, the achievement will be a "dynamic" 
culture of information within the partner system in the 
encounter. Of course, all systems to be so called, have to have 
within them patterns of communication and control, that is, a 
culture of information of some sort. Therefore, all projects, 
programs and campaigns of literacy and post-literacy will already 
be cultures of information. What we are hoping for is that this 
will be a renewed culture, a more dynamic culture, with new norms 
and new habits about the generation, storage, retrieval and use 
of information. It will be a culture of information that is not 
merely efficient but moral. Thus, it will be a culture of 
information that reviews both instrumental questions and moral 
questions, that evaluates both individual achievements and social 
disparities. Within the system boundaries, it will be a 
democratic culture wherein information flows within the system up 
and down and is accessible to all concerned. 
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Figure 1 : A Model for Evaluation Planning and Management in the Context of 
Program Implementation and Policy Assessment 
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